Flash optimization during retinal burst imaging

ABSTRACT

An apparatus for imaging an interior of an eye includes a light sensitive sensor, a plurality of light emitters (LEs) capable of outputting light, a plurality of nonvisible light emitters (NV-LEs) capable of outputting nonvisible light, and a controller. The controller is coupled to the plurality of LEs, the plurality of NV-LEs, and the light sensitive sensor, and the controller implements logic that when executed by the controller causes the apparatus to perform operations. The operations include illuminating the eye with the nonvisible light from the plurality of NV-LEs, and determining an amount of reflection of the nonvisible light from the eye for each of the NV-LEs in the plurality of NV-LEs. The operations also include illuminating the eye with selected one or more of the LEs in the plurality of LEs, and capturing, with the light sensitive sensor, a sequence of images of the interior of the eye while the eye is illuminated with the light from the LEs.

REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Application No. 62/573,324,filed on Oct. 17, 2017, the contents of which are incorporated herein byreference.

TECHNICAL FIELD

This disclosure relates generally to imaging technologies, and inparticular, relates to retinal imaging.

BACKGROUND INFORMATION

Retinal imaging is a part of basic eye exams for screening, fielddiagnosis, and progress monitoring of many retinal diseases. A highfidelity retinal image is important for accurate screening, diagnosis,and monitoring. Bright illumination of the posterior interior surface ofthe eye (i.e., retina) through the pupil improves image fidelity whileoften creating optical aberrations or image artifacts, such as lensflare. Lens flare is a phenomenon where light scatters off of interiorcomponents of a lens system due to internal reflections, refractiveindex changes at various internal boundaries, imperfections, orotherwise. This scattered light shows up in the retinal image as lensflare, which is deleterious to the image quality. The brighter theillumination, the more pronounced the lens flare, which undermines thegoal of improving image fidelity. Other image artifacts may arise due tocorneal reflections or iris reflections from misalignment with thepupil.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. Not all instances of an element arenecessarily labeled so as not to clutter the drawings where appropriate.The drawings are not necessarily to scale, emphasis instead being placedupon illustrating the principles being described.

FIG. 1A illustrates a system for imaging an interior of an eye, inaccordance with an embodiment of the disclosure.

FIG. 1B illustrates a frontal view of a dynamic illuminator included inthe apparatus of FIG. 1A, in accordance with an embodiment of thedisclosure.

FIG. 1C illustrates a frontal view of a dynamic illuminator included inthe apparatus of FIG. 1A, in accordance with an embodiment of thedisclosure.

FIG. 2 is a diagram illustrating a demonstrative retinal imaging systemusing a dynamic illuminator, in accordance with an embodiment of thedisclosure.

FIG. 3 is a functional block diagram of a retinal camera including anintegrated image signal processor, in accordance with an embodiment ofthe disclosure.

FIG. 4 is a block flow diagram illustrating image processing by aretinal camera including an integrated image signal processor, inaccordance with an embodiment of the disclosure.

FIGS. 5A-5C illustrate image frames of a retina captured with differentillumination patterns, in accordance with an embodiment of thedisclosure.

FIG. 6 illustrates focus stacking images of an iris, in accordance withan embodiment of the disclosure.

FIG. 7 illustrates a flow chart for a method of imaging an interior ofan eye, in accordance with an embodiment of the disclosure.

DETAILED DESCRIPTION

Embodiments of a system, apparatus, and method of flash optimizationduring retinal burst imaging are described herein. In the followingdescription numerous specific details are set forth to provide athorough understanding of the embodiments. One skilled in the relevantart will recognize, however, that the techniques described herein can bepracticed without one or more of the specific details, or with othermethods, components, materials, etc. In other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obscuring certain aspects.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

High fidelity retinal images are important for screening, diagnosing,and monitoring many retinal diseases. To this end, reducing oreliminating instances of image artifacts (e.g., deleterious cornealreflections, etc.) that occlude, or otherwise malign portions of theretinal image is desirable. Embodiments described herein describesystems and methods for reducing ocular reflection when imaging theinterior of an eye. The images captured are combined to form a compositeimage of the interior of the eye (e.g., iris). Using images that havefewer defects reduces the number of images that need to be captured, andalso reduces the processing power needed to produce a quality compositeimage.

In some embodiments, a modular printed circuit board (PCB) mount may beused for an illumination assembly to image the eye. Multiple visibleflash sources may be mounted on the PCB and it may be necessary quicklydetermine which lights should fire (e.g., turning on the light sourcefor a short temporal duration, and then turning off the light source).Firing the visible flash sources either sequentially or randomly willlikely give at least one good frame (e.g., an in-focus, well-exposed,frame), but will also provide a high fraction of bad frames as well(e.g., out of focus, poorly exposed frames). Here the order and numberof flash sources fired to capture image frames is optimized usinginfrared (IR) pre-illumination and monitoring of respective reflectedlight from the different positions of IR pre-illuminations.

In several embodiments, multiple IR sources may be located on the PCB asclose as possible to the white/visible sources. For example, the housingof each IR source can be in contact with the housing of eachwhite/visible light source, or the IR light and the visible light areemitted from the same diode structure. All of the IR sources may befired sequentially or randomly, and a controller (e.g., general purposeprocessor and memory, distributed system, image signal processor, or thelike) may determine which of the IR sources produced the fewestreflections and other image defects. The IR reflectance may becorrelated to visible (e.g., white light or the like) reflectance. Basedon the IR reflections observed, the controller determines which visibleflash sources should be fired during visible image capture.

In some embodiments, the image sensor with image processing may be usedto detect the IR light reflected back from the eye, and the controllermay determine which visible light to fire based on the IR reflectionsobserved by the camera. However, in another or the same embodiment, acollection of photodiodes, located proximate to the visible and IR lightemitters, may be used to detect low-reflection frames. Because in someembodiments it is not desirable to capture just the darkest frame (whichmay be poorly illuminated) it is possible to use color filters on thephotodiodes and detect the brightest, mostly-red (since retinas arepredominantly red), image. This could be accomplished extremely fastduring capture, and potentially better accommodate moving eyes. In someembodiments, the an analog circuitry can control the sequential firingof the IR light emitters, analyze the outputs of the photodiodes in eachstep of the sequence, and control the firing of the visible lightemitters positioned around the image path according to the analysis. Inother or the same embodiments, an additional microprocessor, or theretinal camera's existing controller, can be used to accomplish thefunctionalities of such an analog circuit. In some embodiments, thephotodiodes are mounted on the same PCB as the visible and IR lightsources. In some embodiments these light detecting photodiodes could belocated near or around the image sensor plane. This might help todetermine if large reflections are getting to the image plane.

In some embodiment, during pre-capture the subject is being aligned withthe optical pathway, and only infrared illumination may be used to imagethe eye and determine where reflections are originating from. IRreflections may be measured with the camera (or a plurality ofphotodiodes) and these signals may be received with a controller whichdetermines the IR light sources that produced the reflections. It isappreciated that the IR pre-illumination may happen sequentially byfiring each IR light emitter in order, or may happen in a random orother order. The image sensor receives images which are illuminated byinfrared, and based on those images (and any other informationavailable) the system can make decisions about how and when to entersubsequent phases. Based on information available in the infraredspectrum, the system will determine a set of white illumination sourcesto sequentially fire which are expected to yield high quality images.This can be accomplished by sequentially firing the available infraredilluminators and determining a quality metric for each. Then, a model ofthe optical system may be used to determine a set of white illuminatorsexpected to yield similarly high quality images. It is appreciated thatthe IR illumination period may be longer or shorter than the visibleillumination period, or vice versa, depending on how long it takes todetermine which visible light sources will produce quality images.

Since the capture phase is of limited duration, it is important to makeany decisions about system configuration in real-time. Any methods whichrely on data gathered by the image sensor may be subject to latencyinduced by data transfer and processing. As stated, one potential methodto update flash sequence in real-time is to use a set of photodiodes toinfer image quality conditions. Misaligned illumination may becharacterized by dark images. Conversely, corneal reflections ofteninvolve bright white regions. High quality retinal images are generallyof medium brightness and primarily red in hue. Photodiodes with lightfilters in the imaging pathway could detect these conditions veryrapidly (in timescales much shorter than the image sensor exposure), andcontinuously modify the set of enabled illuminators to optimize forfavorable conditions.

Capture may begin when visible LEDs (e.g., white LEDs or otherillumination sources such as laser diodes, flash bulbs or the like) inthe retinal imaging system turn on. The bright light causes thesubject's iris to rapidly close. Accordingly, there is a narrow windowof opportunity to gather data which will be later used to construct aretinal image. During this phase it is important to control the systemto collect the best data possible. Post-capture begins once the whitelight has turned off. In this phase, there is time to process the datacollected by the system during previous phases.

The following disclosure will describe the embodiments discussed aboveas they pertain to FIGS. 1A-7.

FIG. 1A illustrates a system 100 for imaging an interior of an eye, inaccordance with an embodiment of the disclosure. The illustratedembodiment of imaging system 100 includes a dynamic illuminator 105, acamera 110, a controller 115, and an optical relay system 120 capable ofcapturing burst image frames of a retina 125 in eye 101. Also shown areiris 135 and pupil 130 of eye 101; light passes through pupil 130 into,and out of, eye 101, and iris 135 may increase or decrease in sizedepending on light conditions.

As will be shown in greater detail in connection with FIG. 1B, dynamicilluminator 105 includes a plurality of visible light emitting diodes(V-LEDs) capable of outputting visible light, and plurality of infraredlight emitting diodes (IR-LEDs) capable of outputting infrared light.Dynamic illuminator 105, its constituent components, and camera 110(which may be capable of capturing >240 frames/s) are coupled tocontroller 115. Controller 115 implements logic that when executed bycontroller 115 causes system 100 to perform a variety of operationsincluding illuminating eye 101 with the infrared light from theplurality of IR-LEDs, and determining an amount (e.g., intensity,intensity with respect to location or the like) of reflection of theinfrared light from eye 101 for each of the IR-LEDs in the plurality ofIR-LEDs. System 100 may then illuminate eye 101 with a selected one ormore of the V-LEDs in the plurality of V-LEDs based at least in part onthe amount of reflection of the infrared light for each of the IR-LEDs.System 100 may then capture, with camera 110, a sequence of images ofthe interior (e.g., retina) of eye 101 while eye 101 is illuminated withthe visible light from the V-LEDs. In other words, in the depictedembodiment, dynamic illuminator 105 may first illuminate eye 101 with IRlight, then camera 110 may image the IR light reflected from eye 101.Controller 115 will process the images of the reflected IR light and seewhere there are reflections that impair image quality. Controller 115may see what IR-LEDs these reflections came from, and also see whichIR-LEDs did not produce a reflection. Controller 115 may then only firethe V-LEDs that are collocated with the IR-LEDs that did not produce areflection (e.g., disabling some of the V-LEDs when the amount ofreflection of the infrared light from a corresponding IR-LED is greaterthan a threshold reflectance value). Controller 115 may also determinean order of the V-LEDs to illuminate eye 101 with, in response to theamount of reflection of the infrared light from the eye for each of theIR-LEDs. Thus, camera 110 captures visible images of eye 101 that mostlydo not contain a reflection.

Since the reflection profile of IR light and visible light from eye 101may not be the same (e.g., because eye 101 may absorb more IR light thanvisible light or vice versa), in some embodiments, controller 115 maycorrelate the amount of reflection of the infrared light from eye 101for each of the IR-LEDs with a second amount of reflection of thevisible light from eye 101 for each of the V-LEDs. In other words, thereflection profile of IR light is correlated to the reflection profileof visible light so the controller knows which V-LEDs to turn on or offafter looking at the reflection profile of the IR light.

It is appreciated that dynamic illuminator 105 may emit visible and IRlight pulses in any order and even in parallel, and capture images in asimilar manner. For example, dynamic illuminator 105 may sequentiallyfire all of the IR-LEDs, then controller 115 can develop a reflectanceprofile for all of the IR-LEDs. Then dynamic illuminator 105 can firethe select V-LEDs. However, other embodiments, dynamic illuminator 105may fire one IR-LED, then one V-LED, etc. In other embodiments, multipleIR-LEDs and V-LEDs may be fired at the same time.

At least some of the visible images captured may be combined to form acomposite image using at least one of focus stacking (i.e., combiningmultiple images taken at different focus distances to give a resultingimage with a greater depth of field than any of the individual sourceimages), image stitching (i.e., combining multiple photographic imageswith overlapping fields of view to produce a segmented panorama orhigh-resolution image), image blending (i.e. combining a backgroundimage and foreground image giving the appearance of partialtransparency), or any combination thereof.

In addition to only firing V-LEDs that are likely to produce images withlow reflection, it is appreciated that other techniques to filter outpoor quality images may be used. For example, a first set of poorquality images may include overexposed images having a luminance value(e.g., an average luminance value across all pixels, or sets of pixels,in the image) greater than a first threshold luminance value, orunderexposed images having a luminance value less than a secondthreshold luminance value. In some embodiments, the images in the firstset may not be clearly resolved for other reasons such as the imagebeing too blurry (e.g., because the image sensor moved during capture),the images not containing an image of the retina (e.g., because thesubject moved during image capture), or the like. Images may be removedvia manual selection or by automated selection (e.g., using highpass/low pass filters to remove images with luminance values that aretoo high or too low, and/or or using a machine learning algorithm toremove images not including a retina, or the like).

FIG. 1B illustrates a frontal view of dynamic illuminator 105 includedin the apparatus of FIG. 1A, in accordance with an embodiment of thedisclosure. Dynamic illuminator 105 includes V-LEDs 151, IR-LEDs 153,and photodiodes 155 (capable of absorbing IR light). In the depictedembodiment, V-LEDs 151, IR-LEDs 153, and photodiodes 155 aresubstantially collocated with one another. Moreover, there are 16 setsof each of V-LEDs 151, IR-LEDs 153, and photodiodes 155 evenly (or inother embodiments unevenly) spaced around ring 160 (e.g., including thePCB discussed above) of dynamic illuminator 105. One of ordinary skillin the art will appreciate that dynamic illuminator 105 may take anumber of shapes (other than a ring) and may include any number ofV-LEDs 151, IR-LEDs 153, and photodiodes 155.

In one embodiment, the controller (e.g., controller 105) may fire allthe IR-LEDs 153 in a clockwise pattern (e.g., 1-16) around dynamicilluminator 105, and then the controller may fire only some of theV-LEDs but similarly in a clockwise pattern (e.g., 1, 3, 5, 6, 8, 9, 11etc.). Alternatively, the order of firing both IR-LEDs 153 and V-LEDs151 may be random. While in some embodiments, the camera may be used todetermine which IR-LEDs 153 produce reflection; as will be explainedblow, in other embodiments, photodiodes 155 (and also the camera) may beused to determine the amount of reflection with less processing power.

Plurality of photodiodes 155 (e.g., GaAs based, Si based, or the like)may also be coupled to the controller (e.g., controller 155), and thecontroller causes dynamic illuminator 105 to perform operationsincluding measuring, with the plurality of photodiodes, the amount ofreflection of the infrared light from the eye. The controller mayanalyze the amount of reflection measured by the plurality ofphotodiodes to determine the amount of reflection of the infrared lightfrom the eye for each of the IR-LEDs. In some embodiments, this maysimply be a threshold level of reflectance, where if one of photodiodes155 receives greater than a threshold level of IR light reflected fromthe eye, the controller will not turn on the corresponding V-LED duringvisible image capture. In some embodiments, photodiodes 155 may havecolor filters (e.g., polymer color filters, metal mesh, or the like)disposed over photodiodes 155 to allow IR and/or red light to pass tothe photodiodes while blocking other wavelengths of light. Thus, otherwavelengths of light that may cause errors in measurement of photodiodes155 are removed.

Similarly, the controller may determine when the interior of the eye isin focus based on the amount of reflection of the infrared light fromthe eye, as measured with the plurality of photodiodes 155. This may beachieved with contrast detection autofocus or other techniques. Usingphotodiodes 155 to determine if the camera is in focus, may cut down onprocessing power required to autofocus. However, in other or the sameembodiments, the controller analyzes the amount of IR reflection asmeasured by the camera to determine the amount of reflection of theinfrared light from the eye for each of the IR-LEDs.

It is appreciated that in some embodiments, collocation may include“average co-location” for the IR and white LED's. Rather thanco-locating the both sources, IR-LEDs 153 are disposed adjacent to theV-LEDs 151 (e.g., around the optical diameter) which effectively acts asthe same space (e.g., a shift in X, Y location may be somewhatcompensated by putting the LED's on the same diameter since the opticalsystem is symmetric). This is depicted in the “alternate diodeconfiguration”. One of ordinary skill in the art having the benefit ofthe present disclosure will appreciate that other diode configurations(not depicted) are contemplated.

One of ordinary skill in the art will appreciate that the diodes (V-LEDsand IR-LEDs) depicted in FIGS. 1A and 1B are just one embodiment of“light emitters” that may be used to illuminate the eye. Other lightsources such as flash bulbs, lasers or the like may be used.Additionally, it is not necessary to use just visible and IR light.V-LEDs may be replaced with nonvisible light emitters (NV-LEs) or otherlight emitters (LEs) that may emit visible light. Also IR-LEDs may bereplaced with other NV-LEs that produce other nonvisible wavelengths oflight (e.g., low-energy ultraviolet, or the like).

FIG. 1C illustrates a frontal view of a dynamic illuminator 105 includedin the apparatus of FIG. 1A, in accordance with an embodiment of thedisclosure. As shown, dynamic illuminator 105 in FIG. 1C has many of thesame components as the dynamic illuminator depicted in FIG. 1B. However,in FIG. 1C, two IR-LEDs 153 are disposed on either side of each V-LED151. Additionally, here, IR-LEDs 153 and V-LEDs 151 disposed in theinner circle may fire first (either all at once or individually),followed by sequential firing of the lines of LEDs disposed outside theinner circle (e.g., line 2, line 3, line 4, etc.). One of ordinary skillin the art will appreciate that the LEDs in the lines depicted may firein any order (e.g., sequentially, randomly, or the like), and that theorder of LED firing described here is merely to illustrate severalexamples. Additionally, it is appreciated that not all of the lightemitters may be disposed on the same Z plane (in and out of the page),for example visible light emitters may be disposed closer to the eyethan the non-visible light emitters, or vice versa.

In the depicted example, a pupil camera (which may be included in thesame camera as camera 110 in FIG. 1A, or a separate discrete camera),may determine the position of the pupil prior to illuminating the eyewith either IR-LEDs 153 or V-LEDs 151. The pupil camera may provideinformation about the location of the pupil to the system/controller,and the system selects the LEDs that are likely to obtain the bestillumination conditions based, at least in part, on the pupil location.Subsequently, the system can illuminate the eye with fewer lightemitters when capturing the images of the interior of the eye.

In some embodiments, while IR-LEDs 153 and V-LEDs 151 are collocated,non-visible light emitters (e.g., IR-LEDs 153) may be mapped to lightemitters (e.g., V-LEDs 151) that are not collocated. For example,reflectance generated by an IR-LED 153 in row 3 may be a betterindication of visible reflectance generated by a V-LED in row 4 (due tothe different focal lengths of the light emitted from the IR-LEDs 153and V-LEDs 151). Accordingly, a lookup table or the like may be used todetermine which non-visible light emitter(s) should be used toilluminate the eye in order to obtain an accurate prediction of visiblereflection from visible light emitters. As stated, in this embodiment,the non-visible light emitters used to illuminate the eye need not becollocated with the visible light emitters.

FIG. 2 is a diagram illustrating a demonstrative retinal imaging system200 using a dynamic illuminator 205, in accordance with an embodiment ofthe disclosure. Retinal imaging system 200 is one possible (morecomplex) implementation of system 100. The illustrated embodiment ofretinal imaging system 200 includes a dynamic radial illuminator 205,retinal camera 210, controller 215, user interface 215, display 220, andan optical relay system that includes lenses 225 and a beam splitter230. System 200 operates in the same manner as described in connectionwith system 100.

A central section 235 of dynamic illuminator 205 is physicallypositioned in the optical path about the field of view (FOV) of retinal225. In some embodiments, the annular region of dynamic illuminator 205operates as a stop to block many off-axis deleterious reflections beforereaching retinal camera 210. The retinal images are passed throughcentral section 235 to retinal camera 210. In addition to reducing imageartifacts due to deleterious reflections from the cornea, the use ofmultiple illumination locations about the annular region of dynamicilluminator 205 also serves to increase the eyebox of system 200. Theeyebox is the region in space where eye 201 can be located and imaged.In some embodiments, all or some of discrete light sources of dynamicilluminator 205 are disposed outside (e.g., peripheral to) a perimeterof the imaging path extending from retina 225 to retinal camera 210. Inother embodiments, one or more of the discrete light sources of dynamicilluminator 205 are disposed inside the perimeter of the imaging path toretinal camera 210.

Beam splitter 230 (or polarizing beam splitter) is positioned to pass aportion of the light of retinal images to retinal camera 210 whilereflecting display light output from display 220 to eye 201. The displaylight may include a fixation target or other visual stimuli to aidretinal alignment during imaging. In some embodiments, beam splitter 230is more transmissive than reflective. In one embodiment, beam splitter230 is approximately 90% transmissive and 10% reflective. Otherreflectance/transmittance ratios may be implemented. Lenses 225 areprovided throughout system 200 to provide image and light focusing inthe optical paths. User interface 215 provides a mechanism to commenceburst image capture. In one embodiment, user interface 215 is a button,touch screen, mouse or the like.

FIG. 3 is a functional block diagram of a retinal camera 300 includingan integrated image signal processor, in accordance with an embodimentof the disclosure. Retinal camera 300 is one possible implementation ofretinal camera 110 (or 210). The illustrated embodiment of retinalcamera 300 includes a two-dimensional sensor array 305, data conversioncircuitry 310, a memory buffer 315, an integrated image signal processor(ISP) 320, and an output port 325.

During operation, two-dimensional image data (e.g., retinal images) isacquired by sensor array 305 and converted from the analog domain to thedigital domain by data conversion circuitry 310. The image data may beacquired at a high frame rate (e.g., 24, 48, 60, 240, 1000 frames persecond) and stored into memory buffer 315. ISP 320 operates on thebuffered retinal image frames to identify useable or defect regions,annotate the regions of interest in the image frames, and/or combine theuseable regions into high quality, composite retinal images.Accordingly, in one embodiment, some of the image processing tasksdescribed above may be off-boarded to ISP 320 from controller 315. ISP320 may be considered a logical subcomponent of controller 315.

FIG. 4 is a block flow diagram illustrating image processing by aretinal camera (e.g., retinal camera 300 of FIG. 3) including anintegrated image signal processor (e.g., ISP 320 of FIG. 3), inaccordance with an embodiment of the disclosure. As illustrated, imageframes 405A-C of a retina are acquired by a sensor array (e.g., sensorarray 305 of FIG. 3) at a high frame rate, converted into the digitaldomain by data conversion circuitry (e.g., data conversion circuitry 310of FIG. 3), and buffered into a memory buffer (e.g., into memory buffer315 of FIG. 3). An image analyzer 410 is executed by the ISP to analyzethe buffered retinal images 405 (a sort of preprocessing) to determinewhich portions of images frames are of sufficient quality and which areof insufficient quality due to unacceptable image artifacts. Forexample, image analyzer 410 may analyze image frames 405 for blurredportions, portions that do not have sufficient contrast to be useful,are washed out, and/or include unacceptable corneal or iris reflections,or lens flare. Image portions that are deemed unacceptable are flaggedunacceptable (e.g., marked or annotated) while image portions that aredeemed acceptable are flagged as such. The image frames are thenregistered to each other (e.g., pixel-to-pixel alignment), cropped to acommon FOV by image registration/cropping module 415, and then combinedby stacking module 420 into a single composite retinal image 425.Stacking module 420 may combine images to generate high dynamic rangeimages. In other embodiments, image frames 405 are simply combinedwithout analysis and/or annotation of the individual image frames. Allimage processing steps and hardware discussed in connection with FIGS. 3and 4 can be considered part of a “controller” in accordance with theteachings of the present disclosure.

FIGS. 5A-5C illustrate image frames of a retina captured with differentillumination patterns, in accordance with an embodiment of thedisclosure. For example, FIG. 5A illustrates an example image frame 505of retina 525 having an image artifact 525 in the upper right quadrantof the image. This may be an image captured by the camera after IRillumination with an IR-LED located proximate to the upper right handcorner of the eye. Image artifact 525 may be a corneal reflection, areflection or obstruction due to iris, lens flare, or otherwise.Accordingly, the upper right quadrant of image frame 505 may be deemedan unacceptable defect region. Thus, the eye may be illuminated by aV-LED that is not located proximate to the upper right hand corner.Correspondingly, the lower right quadrant of image frame 510, whichincludes image artifact 530, may be deemed to be a defect region. Imageframe 510 may have been illuminated with an IR-LED located proximate tothe bottom right-hand corner. Accordingly, the eye will not beilluminated with the corresponding V-LED. Image frame 515 appears tohave no defects. Image frame 515 may have been illuminated by an IR-LEDdisposed proximate to the lower left-hand corner of the eye.Accordingly, the controller may instruct the V-LEDs in the lower righthand corner of the eye to flash while capturing the visible images ofthe eye.

FIG. 6 illustrates focus stacking images of an iris, in accordance withan embodiment of the disclosure. As shown, four image frames (605A-605D)of a retina are captured with an image sensor. Long lines representfully resolved veins and other anatomical structures in/on the retina;short dashed lines represent out-of-focus or washed out portions of theimage. As shown, the lower left hand corner of image frame 605A is fullyresolved, but the rest of the image is not. Similarly, the middleportion (extending from the upper left-hand corner of the frame to thebottom right-hand corner) of image frame 605B is in focus and fullyresolved, but the rest of image frame 605B is not. The upper right-handcorner of image frame 605C is in focus, but the rest of the image isnot. Lastly, image frame 605D is out of focus and contains no usefulinformation. Accordingly, image frame 605D is removed, and not sent tostacking module 620 for use in composite image 625. The rest of imageframes 605A-605C are sent to stacking module 620 to be combined into asingle high-resolution composite image 625 with a large depth of field.In one embodiment, images may be combined using edge detection, featuredetection, or Fourier analysis.

FIG. 7 illustrates a flow chart for a method 700 of imaging an interiorof an eye, in accordance with an embodiment of the disclosure. It isappreciated that blocks (e.g., blocks 701-707) in method 700 may occurin any order and even in parallel. Moreover, blocks maybe added to, orremoved, from method 700 in accordance with the teachings of the presentdisclosure.

Block 701 shows illuminating the eye with infrared light from infraredlight emitting diodes (IR-LEDs) in a plurality of IR-LEDs. This mayoccur by sequentially turning on and off IR-LEDs. Alternatively, groupsof IR-LEDs may be turned on and off simultaneously. One of ordinaryskill in the art will appreciate that IR-LEDs may be turned on and offin any order and even in parallel. Moreover, the eye may be illuminatedby both IR-LEDs and V-LEDs at the same time.

Block 703 illustrates determining, with a controller, an amount ofreflection of the infrared light from the eye for each of the IR-LEDs.In one embodiment determining may include measuring the light reflectedback (e.g., how much IR light is reflected back to the image sensor,relative to the other IR-LEDS, using saturation level of camera orphotodiodes). The controller may correlate the amount of reflection ofthe infrared light from the eye for each of the IR-LEDs with a secondamount of reflection of the visible light from the eye for each of theV-LEDs. In other words, the controller may know what quantity of IRreflection corresponds to a specific level of visible reflection (e.g.,via a look-up table or the like). In some embodiments, the eye may beilluminated with the infrared light before illuminating the eye with thevisible light; however, in other embodiments the opposite may be true.

In some embodiments, the system may include a plurality of photodiodesand the plurality of photodiodes measure the amount of reflection of theinfrared light from the eye for each of the IR-LEDs. Each of thephotodiodes in the plurality of photodiodes may be collocated with eachof the IR-LEDs and each of the V-LEDs.

Block 705 discloses illuminating the eye with visible (e.g., whitelight, blue light, green light, red light, or any combination thereof)light from visible light emitting diodes (V-LEDs) selected from aplurality of V-LEDs based on the amount of reflection of the infraredlight for each of the IR-LEDs. In some embodiments, illuminating the eyewith visible light from the visible light emitting diodes (V-LEDs)includes disabling some of the V-LEDs when the amount of reflection ofthe infrared light from a corresponding IR-LED is greater than athreshold reflectance value Moreover the controller may determine anorder of the V-LEDs to illuminate the eye with, based on the amount ofreflection of the infrared light. The order and timing of when V-LEDsare activated may be used to further mitigate reflections in the images.

Block 707 shows capturing, with a camera, a sequence of images includingimages of the interior of the eye while the eye is illuminated with thevisible light from the V-LEDs. In some embodiments the system maycontinue flash optimization after the V-LEDs start firing. In someembodiments, either the photodiodes or camera may be used to capturereflected IR light and determine that the camera is in focus to capturethe sequence of images using contrast detection autofocus. In someembodiments this may include measuring, with the controller, anintensity difference of the infrared light between the photodiodes, oran intensity difference of the infrared light between adjacent pixels inthe camera, depending on if the camera or the photodiodes are used toautofocus the camera.

After capturing the series of images, at least some of the images in thesequence of images may be combined using the controller to form acomposite image of the interior of the eye, and combining includes atleast one of focus stacking, image stitching, image blending, or anycombination thereof.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible machine-readable storage medium includes any mechanism thatprovides (i.e., stores) information in a non-transitory form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, manufacturing tool, any device with a set of one or moreprocessors, etc.). For example, a machine-readable storage mediumincludes recordable/non-recordable media (e.g., read only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. An apparatus for imaging an interior of an eye,comprising: a light sensitive sensor; a plurality of light emitters(LEs) capable of outputting visible light; a plurality of nonvisiblelight emitters (NV-LEs) capable of outputting nonvisible light; and acontroller coupled to the plurality of LEs, the plurality of NV-LEs, andthe light sensitive sensor, wherein the controller implements logic thatwhen executed by the controller causes the apparatus to performoperations including: illuminating the eye with the nonvisible lightfrom the plurality of NV-LEs; determining an amount of reflection of thenonvisible light from the eye for each of the NV-LEs in the plurality ofNV-LEs; selecting one or more of the LEs in the plurality of LEs basedat least in part on the amount of reflection of the nonvisible light foreach of the NV-LEs; illuminating the eye with the visible light from theselected one or more of the LEs in the plurality of LEs; and capturing,with the light sensitive sensor, a sequence of images of the interior ofthe eye while the eye is illuminated with the visible light from theselected one or more of the LEs.
 2. The apparatus of claim 1, whereineach of the LEs is collocated with each of the NV-LEs, and wherein thecontroller further implements logic that when executed by the controllercauses the apparatus to perform operations including: correlating theamount of reflection of the nonvisible light from the eye for each ofthe NV-LEs with a second amount of reflection of the visible light fromthe eye for each of the LEs.
 3. The apparatus of claim 2, whereinilluminating the eye with a selected one or more of the LEs includesdisabling some of the LEs when the amount of reflection of thenonvisible light from a corresponding NV-LE is greater than a thresholdreflectance value.
 4. The apparatus of claim 1, further comprising aplurality of photodiodes coupled to the controller, wherein thecontroller further implements logic that when executed by the controllercauses the apparatus to perform operations including: measuring, withthe plurality of photodiodes, the amount of reflection of the nonvisiblelight from the eye, wherein the controller analyzes the amount ofreflection measured by the plurality of photodiodes to determine theamount of reflection of the nonvisible light from the eye for each ofthe NV-LEs.
 5. The apparatus of claim 4, wherein the controller furtherimplements logic that when executed by the controller causes theapparatus to perform operations including: determining when the interiorof the eye is in focus based on the amount of reflection of thenonvisible light from the eye, as measured with the plurality ofphotodiodes.
 6. The apparatus of claim 1, wherein the controller furtherimplements logic that when executed by the controller causes theapparatus to perform operations including: measuring, using the lightsensitive sensor, the amount of reflection of the nonvisible light fromthe eye, wherein the controller analyzes the amount of reflectionmeasured by the light sensitive sensor to determine the amount ofreflection of the nonvisible light from the eye for each of the NV-LEs.7. The apparatus of claim 6, wherein the controller further implementslogic that when executed by the controller causes the apparatus toperform operations including: determining when the interior of the eyeis in focus based on the amount of reflection of the nonvisible lightfrom the eye, as measured by the light sensitive sensor.
 8. Theapparatus of claim 1, wherein the controller further implements logicthat when executed by the controller causes the apparatus to performoperations including: combining at least some of the images in thesequence of images to form a composite image of the interior of the eye,wherein combining includes at least one of focus stacking, imagestitching, image blending, or any combination thereof.
 9. The apparatusof claim 1, wherein illuminating the eye with the nonvisible light fromthe NV-LEs includes sequentially illuminating the eye with each of theNV-LEs in the plurality of NV-LEs, and wherein illuminating the eye withthe visible light from LEs includes sequentially illuminating the eyewith some of the LEs in the plurality of LEs.
 10. The apparatus of claim9, wherein the controller further implements logic that when executed bythe controller causes the apparatus to perform operations including:determining an order of the LEs to illuminate the eye with, in responseto the amount of reflection of the nonvisible light from the eye foreach of the NV-LEs.
 11. The apparatus of claim 1, wherein the pluralityof LEs and the plurality of NV-LEs are disposed on a ring, and whereineach of the LEs and each of the NV-LEs are disposed adjacent to eachother and evenly spaced around the ring.
 12. A method of imaging aninterior of an eye, comprising: illuminating the eye with nonvisiblelight from nonvisible light emitters (NV-LEs) in a plurality of NV-LEs;determining, with a controller, an amount of reflection of thenonvisible light from the eye for each of the NV-LEs; selecting lightemitters (LEs) from a plurality of LEs based on the amount of reflectionof the nonvisible light for each of the NV-LEs; illuminating the eyewith visible light from selected ones of the LEs; and capturing, with alight sensitive sensor, a sequence of images including images of theinterior of the eye while the eye is illuminated with the visible lightfrom the selected ones of the LEs.
 13. The method of claim 12, furthercomprising correlating, with the controller, the amount of reflection ofthe nonvisible light from the eye for each of the NV-LEs with a secondamount of reflection of the visible light from the eye for each of theLEs.
 14. The method of claim 13, wherein illuminating the eye with lightfrom the LEs includes disabling some of the LEs when the amount ofreflection of the nonvisible light from a corresponding NV-LE is greaterthan a threshold reflectance value.
 15. The method of claim 12, furthercomprising determining an order of the LEs to illuminate the eye with,based on the amount of reflection of the nonvisible light.
 16. Themethod of claim 12, further comprising combining, with the controller,at least some of the images in the sequence of images to form acomposite image of the interior of the eye, wherein combining includesat least one of focus stacking, image stitching, image blending, or anycombination thereof.
 17. The method of claim 16, wherein the compositeimage includes an image of the eye's retina.
 18. The method of claim 12,wherein illuminating the eye with the nonvisible light occurs beforeilluminating the eye with the visible light, and wherein each of theNV-LEs is collocated with each of the LEs.
 19. The method of claim 12,further comprising measuring, with a plurality of photodiodes, theamount of reflection of the nonvisible light from the eye for each ofthe NV-LEs.
 20. The method of claim 19, further comprising determiningthat the light sensitive sensor is in focus to capture the sequence ofimages using contrast detection autofocus.
 21. The method of claim 19,wherein measuring the amount of reflection of the nonvisible lightincludes filtering wavelengths of light other than the nonvisible lightwith color filters disposed over the plurality of photodiodes.
 22. Themethod of claim 20, wherein using contrast detection autofocus includesmeasuring, with the controller, an intensity difference of thenonvisible light between the photodiodes.
 23. The method of claim 20,wherein using contrast detection autofocus includes using the controllerto determine an intensity difference of the nonvisible light betweenadjacent pixels in the light sensitive sensor.
 24. The method of claim12, further comprising: capturing an image of a pupil of the eye priorto selecting the LEs from the plurality of LEs; and determining with thecontroller a location of the pupil using the image or the pupil, andwherein selecting the LEs includes using, at least in part, the locationof the pupil to select the LEs.
 25. The method of claim 12, wherein atleast some of the selected LEs are not collocated with each of theNV-LEs used to illuminating the eye.